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ABSTRACT 

The relatively little amount of time that some college students spend reading their textbooks outside of the 
classroom presents a significant threat to their academic success. Using Prenksy’s (2001) digital game-based 
learning (DGBL) principles and Astin’s student involvement theory as frameworks, the purpose of this true 
experiment was to determine whether a significant difference in engagement, as indicated by mental effort and 
time on task, existed between college students who used a digital game-based textbook and students who used a 
traditional print-based textbook. A customizable digital game-based textbook designed using DGBL principles 
in a popular gaming genre embedded with textbook content identical to content in a traditional print-based 
textbook was used. Fifty-four undergraduate college students were randomly assigned to experimental and 
control groups. The results showed a statistically significant, Hotelling’s T 2 (2,52)=25.11, p<. 001, D 2 =1.86 
difference in engagement between participants in the experimental and control groups and a large overall effect 
size. Post hoc analyses indicated the digital game-based textbook group, exerted significantly greater mental 
effort (t = 2.38, p<. 001, d=. 65) and spent significantly more time on task (t = 4.61, p < .025, d= 1.25) than the 
traditional print-based textbook group. 
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INTRODUCTION 

Although a number of students are not compelled to engage in reading the college textbook outside of the 
classroom (Arum & Roska, 2011; Culver & Morse, 2012; Yonker & Cummins-Sebree, 2009), research indicates 
that some college students are very willing to engage in playing videogames outside of the classroom (Alsagoff, 
2005; Moshirnia, 2007). In fact, researchers found that college students can spend as many as 10,000 hours 
playing video/computer games by the time they graduate (Pivec, 2009; Prensky, 2003; Riegle, 2005). If 
researchers can identify a digital game format for textbook content that is compelling to students for 
entertainment purposes and simultaneously increases student engagement with academic content outside of the 
classroom, then there would be a viable learning tool that may be effective for learning. 

In the educational environment, digital games, as a learning tool, have gained very little headway. Although 
many educators do agree that learning should be interesting and fun, they are very apprehensive about including 
digital games as a part of the course curriculum (Gros, 2007), because there is a misconception that games 
cannot be used for learning (Hirumi, Appleman, Rieber, & Van Eck, 2010). At the same time, many education 
institutions are realizing that a number of students are not performing well on the variety of assessments 
designed to evaluate students’ mastery of curriculum content and/or student progress (Arum & Roska, 2011). 
What appears to be missing is research on whether the digital game-based approach truly engages learners in the 
process; information that might compel faculty to reconsider it as an option. Although modem textbooks may be 
an efficient learning tool, some modem textbooks appear to be failing to elicit the interest of college students. 
According to Astin (1999), “The theory of student involvement argues that for a particular curriculum, to 
achieve the effects intended, must elicit sufficient student effort and investment of energy to bring on the desired 
learning and development” (p. 522). Astin’s (1999) comment begs the question as to why colleges and 
universities are not demanding a textbook design that would elicit student engagement, particularly outside the 
course. 

Research has demonstrated (Stratton, 2011), many students are not engaging sufficiently, and in some cases not 
engaging at all with the college textbook, outside of the classroom. Many students are disinterested in reading 
their college textbooks outside of class, despite the fact that the college textbook is typically the primary learning 
resource that students are expected to use outside of the classroom (Lord, 2008). If traditional textbooks are not 
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eliciting mental effort and time on task, educators should be focusing on methodologies that do. One such 
methodology may be digital game-based learning. Digital game-based learning (DGBL) theory emerged as the 
result of the contributions of several scholars focused on this area (Dziomy, 2005), and in particular Prensky 
(2001). Prensky’s (2001) book. Digital Game-Based Learning, introduced the concept of DGBL. Essentially, the 
premise behind DGBL is about merging game design with instructional design. “A DGBL game should feel just 
like a video or computer game, all the way through. But the content and context will have been cleverly designed 
to put you [the student] in a learning situation about some particular area or subject matter” (Prensky, 2001, p. 
146). According to Prensky, DGBL involves an educational game located online or on a computer. DGBL is 
about using key elements, like fun and interactivity, to generate continuous engagement for students 
accompanied by learning of educational content. Prensky (2001) noted that there are six characteristics of 
compelling digital games: a) incorporation of goals and objectives, b) representation/story, c) rules, d) 
interaction, e) outcomes and feedback, and f) challenge in the form of competition, conflict, or opposition. 

Digital game-based learning theory served as the theoretical framework for using a customizable digital game- 
based textbook, which was a customizable digital-game in a popular gaming genre designed using DGBL 
principles. Astin’s (1999) student involvement theory served as the guiding theoretical framework for 
understanding student engagement and developing a hypothesis about students’ mental effort and time on task. 
The purpose of this study was to detennine whether there was a significant difference in student engagement 
with textbook content as indicated by mental effort and time on task, based on the textbook fonnat. In this study, 
the efficacy of using a digital game-based textbook as an alternative to the traditional print-based textbook for 
increasing mental effort and time on task with college textbook content was examined. For a sample of 
undergraduate college students, are there significant differences in engagement as indicated by mental effort and 
time on task, based on textbook format (traditional or digital game-based)? It was hypothesized that there would 
be a significant difference in engagement as indicated by mental effort and time on task with students using a 
digital game-based textbook exerting significantly more mental effort and time on task than students using a 
traditional print-based textbook. 

METHOD 

A convenience sample of 54 participants ranging in age from 18 to 57 was used. The mean age for participants 
in this study was 24.83. Most participants were between the ages of 18 to 24 (n= 38). Non-traditional aged 
students included students between the ages of 25 and 34 (n= 9), as well as students between the ages of 35 and 
60 («=7). Participants were degree-seeking college students at a public community college. Twenty-six of 54 
students were first-year/freshman college students. Twenty-eight were second-year/sophomore college students. 

An ethnically diverse sample was obtained including African-Americans (n=24). Blacks (not American) (n= 6), 
European-Americans (w=5), other (n=5), Asian-Americans (n= 4), Hispanic Americans (n=4), Asians (not 
American) (n— 2), Hispanics (not American) (n= 2), and multiracial students (n= 2). Participants identified as 
female (n = 30), male (n = 23), and other («= 1). 

Materials 

Digital game-based textbook, a customizable PC-based digital game (Thomas, 2017) designed using DGBL 
principles in a popular gaming genre embedded with textbook content identical to content in a traditional print- 
based textbook was used. The digital game is a live-action sequence game with nine levels. The live action 
sequence game, a twitch game, included 100% of the content from the Introduction to Social Science Research 
chapter in the Social Science Research Methods textbook. 

Traditional print-based textbook, one chapter, Introduction to Social Science Research, from Social Science 
Research Methods (Gibbs, 2013) was used. The print-based textbook chapter (see Appendix A 1 was 8.5 x 11 in 
(215.9 x 279.4 mm) in dimension and included 37 pages with four major content areas. In addition to four 
sections and a summary, the textbook chapter included chapter definitions and chapter questions. The chapter 
questions section includes 32 multiple-choice questions with eight questions for each major content area. 

Instrumentation 

Mental effort scale, the Mental Effort Scale or MES (Paas, 1992), the most widely used measure in this area of 
research (Paas, Tuovinen, Tabbers, & Van Gerven, 2003) was used due to its psychometric soundness (Paas, 
1992; Paas & Van Merrienboer, 1994; Van Gog, Paas, & Van Merrienboer, 2008; Kester, Kirschner, & Van 
Merrienboer, as cited in Van Gog et al., 2008) and ease of use. The MES utilizes five 9-point Likert-type items. 
Scores are computed by summing participant responses for each item on the measure, with higher scores 
reflecting greater mental effort expenditure and lower score reflecting lower mental effort expenditure (Paas & 
Van Merrienboer, 1994). 
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Stopwatch, time on task was measured using researcher observation using a stopwatch. The computer screen of 
each activity session was recorded, so that each participant’s time on task could be retrospectively assessed by 
viewing the recording after all data was collected. Questionnaire, a brief six-item questionnaire that captured 
demographic data on participant characteristics was used. The questionnaire content includes questions about 
age, gender, race/ethnicity, college level, and college work completed. Screener, an eligibility screener 
containing 5 questions was used to exclude any prospective participant who was under 18, not enrolled as a 
degree-seeking college student, or beyond the second year of college. 

Procedure 

Setting, the research study was conducted in a controlled setting. A computer lab with individual workstations 
on a college campus was reserved for multiple dates across a two-month time period. Each computer lab used for 
textbook activity sessions had individual workstations labeled with ID numbers. 

Random assignment, a strategy of random assignment without replacement was used to assure an equal number 
of participants in both groups. Twenty-seven slips with the name dgbg (digital game-based textbook group) and 
27 slips with the name pbtg (print-based textbook group) were placed in small sealed envelopes. Fifty-four 
participants were randomly assigned to the digital game-based textbook group or traditional print-based textbook 
group. Each participant drew an envelope on arrival, thus assuring random assignment to conditions of the 
independent variable. 

Textbook activity sessions, after participants were seated at an individual workstation, participants completed an 
online informed consent process. Upon completion of the informed consent process, participants were instructed 
to minimize the online informed consent form. Next, participants were given written instructions, which marked 
the beginning of the textbook activity session. The standard activity session protocol was followed. There was 
no minimum period of time for the textbook activity session. Participants were limited to a maximum period of 
time for study participation, which was 2 hours. When a participant indicated that he or she was finished with the 
activity session, the participant was given the Mental Effort Scale and demographic survey. 

Recording textbook activity sessions, for all textbook activity sessions, digital game-based textbook and print- 
based textbook, screen recording was conducted. The computer screen of each activity session was recorded. 
When the activity session ended, the activity session data was saved with the ID number of the participant so 
that each participant’s time on task could be assessed by viewing the recording after all data was collected. 

Protection of participants, the four primary areas of ethical concern in social research were addressed. In this 
study 1) no harm came to research participants, 2) informed consent was utilized, 3) no invasion of research 
participants’ privacy occurred, and 4) deception was not be used at any point in the study. The participants in 
this study were not marginalized in any way. Approval from the related higher education institutions’ 
Institutional Review Boards was obtained, prior to data collection. No participants from vulnerable populations 
were used in this study. After each activity session concluded, participants who were not in the digital game- 
based group were given the opportunity to view and engage in the digital game-based textbook activity. There 
were no reports of adverse effects from participation in this study. 

Research participation incentive, each participant received a $10.00 gift card as an incentive for research 
participation. 

RESULTS 

The results of a Hotelling’s T 2 test indicated that there was an overall statistically significant difference (see 
Tables 1 and 2) in the mental effort and time on task of participants in the digital game-based textbook group 
and participants in the traditional print-based textbook group, T\ 2, 52) = 25.11 ,p < .001, D 2 =1.86. 


Table 1: Hotelling’s T 2 for mental effort and time on task 


Dependent Variables 

I 2 

DF1 

DF2 

p value 

D 2 

Mental Effort and Time on Task 

25.11 

2 

52 

.00 

1.86 


The multivariate measure of effect size, Mahalanobis distance, D 2 , was computed for overall effect size. A large 
effect size, D 2 =1.86, was found. Like Cohen's d, recommendations for small, moderate, and large effect size 
indices have been provided (Ferron, Hess, Hogarty, & Kromrey 2004; Stevens, 1980; Stevens 2004), with .25, 
.64, and > 1.0 representing small, moderate, and large effect sizes respectively (Stevens, 1980; Stevens, 2002). 
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Post hoc analyses of the means (see Table 2) with univariate t tests (see Table 3) were conducted to determine 
whether the overall significant difference was true for both dependent variables, mental effort and time on task. 
To control for Type I error resulting from post hoc multiple comparisons, a Bonferroni correction (Field, 2009) 
was applied. The adjustment was used with the alpha level of .05, which was divided by the number of post hoc 
tests resulting in an alpha of .025 being used to assess each individual t-test. 


Table 2: Mental effort and time on task by intervention group 



Mental effort 



Time on task 




Group 

Mean 

SD 

Min 

Max 

Mean 

SD 

Min 

Max 

N 

Digital game 

17.33 

5.95 

3 

27 

87.66 

33.22 

5 

124 

27 

Print-based 

13.93 

4.46 

3 

27 

48.22 

29.57 

3 

115 

27 


The results of the post hoc analyses demonstrated a statistically significant difference exists between the digital 
game-based textbook group and print-based textbook group for the dependent variable mental effort (t = 2.38, 
p< .025). A medium effect size, d=. 65, was found. Mental effort was significantly higher for the digital game- 
based textbook group. The result of the reliability analysis yielded a Cronbach’s alpha of .76, which indicates 
that the MES was a reliable measure of mental effort for participants in this study. 

The results of the post hoc analyses (see Table 3) also showed that a significant difference between the digital 
game-based textbook group and print-based textbook group exists for the dependent variable time on task (t = 
4.61, p < .001). A very large effect size, d= 1.25, was found. Time on task was significantly higher for the digital 
game-based textbook group. 


Table 3: Post hoc tests and effect sizes for dependent variables 


Variable 

t 

p value 

Cohen’s 

d 

Mental Effort 

2.38 

.02 

.65 

Time on Task 

4.61 

.00 

1.25 


DISCUSSION 

It is important for educators who select textbooks to know that college students exerted significantly more 
mental effort and time on task (/?<.001) when using a digital game-based textbook designed using DGBL 
principles than when using a traditional textbook. Interestingly, a large overall effect size (D 2 =1.86) was found, 
although the statistical power achieved for this study was only .50. The medium effect size (t = 2.38, p<. 025) 
found for mental effort is compelling, but the very large effect size (t = 4.61 ,p < .001, d= 1.25) found for time on 
task is undeniably convincing evidence of the ability of the ability of a DGBL video game textbook to increase 
student engagement with academic content. 

Like the Sward, Richardson, Kendrick, and Maloney (2008) and Um, Plass, Hayward, & Homer’s (2012) 
studies, this study found a significant difference in the amount of mental effort of participants in the 
experimental and comparison group with mental effort exerted by students being significantly (p<. 025) greater 
for the digital game-based group (M=17.33) than the traditional print-based textbook group (M=13.93). In 
contrast to students in the traditional print-based textbook group who spent an average of 48.22 minutes on the 
textbook activity session, students in the digital game-based textbook spent an average of 87.66 minutes. 

The findings related to time on task match those of Sward et al. (2008), who found that participants in a web- 
based gaming group spent significantly more time on task than participants in a web-based flash-card group; 
Adams, Mayer, MacNamara, Koenig, and Wainess (2012), who found that a narrative game-based group of 
college students spent more time on task when using a modified version of the popular Half-Life 3D digital 
game than a slide-show presentation of the same content; and (Um et al., 2012), who found that participants in 
the positive emotional design group spent more time on task than participants in a neutral group when engaged 
with academic content presented in an interactive multimedia format. 

There are several limitations for this study. The scope of this study was limited to the conceptual frameworks of 
mental effort and time on task, the key concepts discussed in the first three premises of Astin’s (1985) student 
involvement theory and did not include learning as a dependent variable. In addition, a convenience sample was 
used in this study. Because a convenience sample is a non-probability sampling design, scientific inferences 
about what exists in the population of interest cannot be made. This shortcoming was largely unavoidable, due to 
the nature of the study. 
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The use of a self-report measure of mental effort prevented a determination of whether or not research 
participants honestly reported their mental effort. However, the nature of the measure did not suggest a need of 
the participants to lie or give socially desirable responses. A final limitation of this study is that only a single 
textbook chapter was used that focuses on one subject (research methods). This focus means that scientific 
inferences about the efficacy of the digital game-based textbook with other subject areas are outside of the scope 
of this study. 

CONCLUSION 

The results of this study show that a digital game-based textbook designed using DGBL principles in a popular 
gaming genre with fun and interactivity as key elements with embedded educational content, is a learning tool 
that caused students to exert significantly greater mental effort and spend significantly more time on task with 
textbook content outside of the classroom. The digital game-based textbook was a superior elicitor of mental 
effort and time on task, because instead of students spending time with a game inside of a learning environment, 
students were spending time with learning inside of a gaming environment. The live-action sequence game, the 
digital-based based textbook, was effective because the game integrates all six of Prensky’s (2001) 
characteristics of compelling digital games: a) incorporation of goals and objectives, b) representation/story, c) 
rules, d) interaction, e) outcomes and feedback, and f) challenge. This study provides stakeholders in higher 
education with research that may lead to a viable alternative textbook format to the traditional print based- 
textbook format. It would be useful for textbook authors and publishers to focus textbook development efforts on 
innovative textbook design of digital game-based textbooks using DGBL principles. 

Although previous studies have been conducted on educational games and their relation to student learning 
(Adams et al., 2012; Alsagoff, 2005; Baek & Heo, 2010; Kiili, 2005, Pivec, 2009), this is the first study to 
address the gap in the literature on digital game-based learning theory and its relation to student involvement, as 
conceptualized by Astin (1985), with college course content using a digital-game designed using DGBL 
principles outside of the classroom. This study adds to our knowledge about the efficacy of alternatives to 
traditional print-based textbooks for student engagement in outside-of-class studying. This study provides robust 
evidence that a customizable digital game-based textbook designed using DGBL principles increases college 
student involvement with textbook content outside of the classroom, which suggests that it may also lead to 
better learning of college course material and ultimately better academic performance of students in college 
courses. 

Recommendations for Further Research 

During data collection, it was observed that some participants in both the digital game-based textbook and 
traditional print-based textbook voluntarily elected to take notes on the academic content that was presented 
throughout the textbook activity session. Although note taking was outside the scope of this study, future studies 
should examine whether or not a significant difference in amount and quality of note taking exists for students 
using a digital game-based textbook and students using a print-based textbook. 

Future studies should also extend this research by including learning as a dependent variable. Astin’s (1999) 
student involvement theory was used as the theoretical framework for this study, and learning is the outcome 
variable in Astin’s (1985) theory, which postulates that increased mental effort and time on task leads to 
increased learning (Astin, 1985, 1999). Although learning was outside the scope of this study, mastery learning 
was strategically designed into the structure of the digital game-based textbook through not allowing participants 
to proceed to the next section without first mastering the current section. The extent to which participants 
engage in mastery learning could be considered in future studies using a learning assessment instrument. 

Finally, it would be useful for researchers to conduct research studies that use digital games in varied formats. 
There are a variety of popular digital game genres (Baek & Heo, 2010). This study used a live-action sequence 
game, which is also known as a twitch game in the entertainment market. Twitch games involve the player’s 
thumbs moving at a very fast pace (Prensky, 2001). Additional research is needed to determine if digital game- 
based textbooks grounded in other popular digital game genres are also effective for increasing student 
engagement with textbook content outside of the classroom. 
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